Yes, you're right.

The Number constructor will call ToNumeric on its argument, which then calls ToNumber on it, which calls ToPrimitive on the object with a hint of number, which results in OrdinaryToPrimitive being called, which does

1. If hint is string, then
  a. Let methodNames be « "toString", "valueOf" ».
2. Else,
  a. Let methodNames be « "valueOf", "toString" ».
3. For each element name of methodNames, do
  a. Let method be ? Get(O, name).
  b. If IsCallable(method) is true, then
    i. Let result be ? Call(method, O).
    ii. If Type(result) is not Object, return result.

The hint is number, so the method names called are first valueOf and then (if that doesn't exist) toString.

valueOf returns the object itself in the case of a plain object, and the array itself in the case of an array, so that's no help - it moves on to toString.

Array.prototype.toString returns the array values joined - arr.toString() is equivalent to arr.join(','). The empty array joins into the empty string. So

Number([])

turns into

Number('')

and so the empty string turns into 0.

But objects, when toString is called on them, give [object Object]

console.log({}.toString());

which cannot be coerced into a number, thus NaN.